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We investigate the influence of finite reso- 
lution on measurement uncertainty from 
the perspective of the Guide to the 
Expression of Uncertainty in Measurement 
(GUM). Finite resolution in a measure- 
ment that is perturbed by Gaussian noise 
yields a distribution of resuhs that strongly 
depends on the location of the true value 
relative to the resolution increment. We 
show that there is no simple expression 
relating the standard deviation of the dis- 
tribution of measurement results to the 
associated uncertainty at a specified level 
of confidence. There is, however, an ana- 
lytic relation between the mean value and 
the standard deviation of the measurement 
distribution. We further investigate the 



conflict between the GUM and ISO 
14253-2 regarding the method of evaluat- 
ing the standard uncertainty due to finite 
resolution and show that, on average, the 
GUM method is superior, but still approxi- 
mate. 
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1. Introduction 

The issue of measurement uncertainty is becoming 
increasingly relevant to both calibration laboratories 
and factory floor metrology In some cases measure- 
ment uncertainty has significant economic impact on 
the cost of a product. For example, the ISO standard 
14253-1 [1] (default condition) requires the expanded 
uncertainty to be subtracted from both ends of the prod- 
uct tolerance yielding a smaller "conformance zone." A 
measurement result must lie in this zone in order for the 
manufacturer to distribute the product. Avoiding over- 
estimation of the measurement uncertainty can result in 
a larger conformance zone and hence lower product 
cost. In some situations the finite resolution of a meas- 
uring instrument is a significant contributor to the 
uncertainty statement. For example, the manufacturers 
of hand held digital calipers typically set the resolution 



of the instrument such that repeated measurements of 
the same quantity yield nearly the same result, within 
one or two units of the least count (i.e., resolution) of 
the instrument. In principle the caliper would be more 
accurate with an additional display digit, however, cus- 
tomers often perceive quality as the ability of the 
instrument to yield the same value for repeated meas- 
urements. The uncertainty budget of a measurement 
performed with a hand held instrument typically con- 
tains only a few significant influence quantities since 
most quantities such as the accuracy of its calibration 
(typically reported at 20 °C) and thermal effects associ- 
ated with the measurement are usually small compared 
to the instrument resolution. Therefore the inclusion (or 
not) of the uncertainty associated with the finite resolu- 
tion of the display can significantly affect the magni- 
tude of the reported uncertainty. Similar issues may 
arise with the finite resolution introduced by analog to 
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digital conversion electronics where repeated sampling 
of a signal differs by only one or two bits. 

In this paper we first examine some general proper- 
ties of the probability distribution associated with a 
measurement recorded with finite resolution and per- 
turbed with Gaussian noise. The distribution of the 
measurement results is generated via computer simula- 
tion so that we can control the underlying population 
mean (true value) and standard deviation (of Gaussian 
noise) and produce large quantities of recorded meas- 
urement results. We then examine how the finite reso- 
lution and measurement noise impacts the uncertainty 
evaluation under different measurement scenarios. 

The Guide to the Expression of Uncertainty in 
Measurement (GUM) [2] identifies the finite resolution 
of a measuring instrument as a contributor to measure- 
ment uncertainty of a measurement from that instru- 
ment. It further suggests that this effect should be eval- 
uated by a (Type B) uniform distribution with a full 
width of one resolution unit, resulting in a standard 
uncertainty of ~^ in units of the resolution (GUM 
F.2.2.1). The GUM also suggests that this contribution 
is uncorrelated to other uncertainty sources and that it 
should be added in a root-sum-of-squares (RSS) man- 
ner in the uncertainty statement (e.g., see GUM H.6). 

Alternatively, ISO 14253-2 [3] recommends examin- 
ing the standard uncertainty of the resolution relative to 
the standard deviation of repeated measurements and 
then including the larger of the two in the uncertainty 
evaluation and discarding the lesser of the two. The 
rationale of this method is that the resolution is already 
intertwined with the standard deviation of repeated 
measurements, since that data is recorded with the res- 
olution of the instrument. 

We summarize these two procedures as: 

Rule 1: RSS the standard uncertainty of the resolution 
(Type B via a uniform distribution) with the 
uncertainty associated with the standard devia- 
tion of repeated measurements in the uncer- 
tainty evaluation. 

Rule 2: Include the larger of the following two: the 
standard uncertainty of the resolution (Type B 
via a uniform distribution) and the standard 
deviation of repeated measurements in the 
uncertainty evaluation and discard the smaller 
of the two. 

Clearly Rule 1 and Rule 2 cannot both simultaneously 
be the best estimate of the uncertainty for all measure- 
ment cases. 



There are two measurement scenarios we will con- 
sider. The "special test" scenario involves constructing 
an uncertainty statement for one specific measurement 
quantity. Typically this will involve repeated observa- 
tions of the quantity, each recorded with finite resolu- 
tion. The best estimate of the measurand is considered 
to be the mean of the repeated observations (after all 
corrections are applied) and the uncertainty statement 
will be associated with this mean value. 

The "measurement process" scenario involves con- 
structing an uncertainty statement that will be applica- 
ble to a series of future measurement results. This is 
typical of commercial metrology where a large number 
of nearly identical artifacts or workpieces are produced 
and only a single measurement, having finite resolu- 
tion, is recorded. In this scenario, the uncertainty state- 
ment is developed once and then associated with each 
future (single observation) measurement result. The 
uncertainty evaluation process will involve, among 
other things, repeated measurements of calibrated arti- 
facts, recorded with finite resolution, and used to char- 
acterize the measurement variation. 

Without loss of generality we consider the case 
where a resolution increment has the value of unity, and 
the infinite resolution value of the measurement, //, lies 
between zero and one-half (all other cases are modulo 
this problem). We consider the case where the measure- 
ment is corrupted by noise from a Gaussian distribution 
with standard deviation ex. The situation of interest will 
be for (7< 1 since large cxis equivalent to infinite reso- 
lution. Our approach will be to examine both the spe- 
cial test scenario and the measurement process scenario 
in the limit of large sample size. 



2. Probability Distributions 

In the limit of large sample size, expectation values 
are estimated by sample statistics. Finite resolution 
requires that only integer values will be observed. 
Hence a continuous probability distribution function 
(pdf) becomes a discrete pdf that can be written: 

Discrete pdf (x) =w{jJi,a,n)5{x- n) 
/2 :-oo,..., -3, -2, -1,0,1,2,3,..., +00 

with w(fi, (J,n)= j pdf (//, CT, x) dx 

= 0(//,CT,/7+^)-0(//,CT,/7-|) (1) 
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where w is the probability mass function, d(x-n) is a 
delta function, and O is the cumulative probability dis- 
tribution function. Figure 1 illustrates the quantization 
(using Eq. (1)) of a Gaussian distribution with jU = 0,4 
and a= I. 



3. Descriptive Statistics: Mean and 
Standard Deviation 

Using the discrete probability distribution, expecta- 
tion values can be calculated. The expected value of the 



mean, x, and the standard deviation, s, can be calculat- 
ed from: 



x(jU,0')= ^ w(fi, a, n)n and 



si/^,a) = J X ^(M'^' n)(n-xy 



(2) 



Figure 2 displays plots of x and s for relevant values of 
jU and a. The plot for x shows the expected step func- 
tion (due to the discrete resolution) at jU = 0,5 when 




(a) 
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(b) 

Fig. 1. (a) A Gaussian distribution with |a = 0.4 and a= 1 together with a comb function 
representing the instrument's finite resolution, (b) The corresponding discrete distribution 
with values at the integer values equal to the probability mass function. 
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(b) 

Fig. 2. Plots of (a) the sample mean, x, and (b) sample standard deviation, s, as a func- 
tion of ^ and a for data recorded with a resolution of unity. 



(7^0, and the expected behavior x ^ jU for large (7, 
Similarly, the plot for s displays the expected rapid rise 
when jU -^ 0.5 and (7 is small, and the expected behav- 
ior 5 ^ (7 for large (7. 

An interesting and intuitive feature of Fig. 2 can be 
more easily seen in Fig, 3, which shows the mean x as 
a function of (7 for various values of jU. For any given 
value of X < 0.5, which is a horizontal line on Fig, 3, 



the possible values of ji are always greater than x. 
(Equivalently stated, for < jU < 0.5 and any a, then 
X < jX). Hence the observed discrete population mean is 
always biased toward the closest resolution increment 
regardless of the value of o (for large o this bias 
becomes insignificant). For small (7, e,g., o< 0.5, it is 
incorrect to believe that uncertainty associated with the 
sample mean approaches zero as a result of averaging a 
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Fig. 3. Plot of X vs. a for various values of /j; note that for any particular x the possible values of /j are always 
greater than x resulting in a bias of i toward the resolution unit. 



large sample; rather it approaches a fixed value result- 
ing in a systematic error. 

Figure 4 shows the dependence ofs on a for various 
values of jU. As c becomes large, s ^ a, approaching it 
from above. The slight overestimation of (7 by ^ is well 
known and the difference is called "Sheppard's correc- 
tion" [4]. We fiirther point out that s > (7 when ^ > ;^ 
regardless of jU. Similarly, ;/ff > c when ^ < ;^ regard- 
less of jU; hence Max[^ , s] > cfor all jU, a fact we will 
make use of later. 



What is not obvious from examining the previous 
plots is the interaction of x and s. In particular, there are 
combinations of x and s that are forbidden. We examine 
this by creating a dense grid of (jU, o) pairs over the 
domain < jU < 0.5 and < (7< 0.5. For each (//, o) we 
create (numerically, with a sample size of 10,000) a 
finite resolution sample of Gaussian distributed values 
and calculate the corresponding mean ( x ) and standard 
deviation (s). Figure 5 shows a plot of the resulting x, s 
space, showing that there is a disk with a radius of one- 
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Fig. 4. Plot of the standard deviation of the finite resolution population as a function of the standard deviation 
of the Gaussian noise. 



147 



Volume 113, Number 3, May- June 2008 

Journal of Research of the National Institute of Standards and Technology 




0.0.0 



(a) 




Fig. 5. (a) the value of i, s, and x-fifor all combinations of < jU < 0.5 and < a< 0.5; 
(b) the 2D plot of the x, s, coordinates shown in (a) using the same color coding for the 
systematic error. 



148 



Volume 113, Number 3, May-June 2008 

Journal of Research of the National Institute of Standards and Technology 



half resolution unit, as described in Eq. (3), inside 
which no combination of x and s values can occur. 



X -\-s 



Ij +"^w(n,fi,a){n' -n)^(\] V//,(7. (3) 



The dense set of x, s coordinates that occur on the disk 
boundary is understood by considering values of n 
where the equality in Eq. (3) holds, i.e., the disk bound- 
ry. Since each term in the infinite sum in Eq. (3) must 
be positive or zero, the equality can only hold when 
infinite sum of the terms >v(//, G, n)(n^ - n) is zero. This 
occurs when the weight functions tv(//, a, 0) + tv(//, a, 
1) = 1 . That is, for any ji(0 < fi< 0.5) and a such that 
0(//, a, ly) - 0(//, (J, -y) = 1 then all of the probability 
is contained in the weight functions w(ji, a, 0) and 
w(fi, (J, 1) and hence the infinite sum of the terms 
w(fi, G, n){n^ - n) is zero. Hence, when the equality of 
Eq. (3) holds the values of x and s satisfy the equation 
of a circle with a radius one half resolution unit. (Since 
all Gaussian distributions have some probability, albeit 
infinitesimal, to extend to infinity, the disk radius is 
actually a limiting value.) 

The Z dimension in Fig. 5 corresponds to the system- 
atic (i.e., expected) error x - ji that is associated with 
each (//, a) point. As previously described, the expect- 
ed error is always biased toward the resolution unit, i.e., 
the bias is negative as shown in the Z coordinate of Fig. 
5. Furthermore, as seen in the figure, it is only the 

values where — x +s^ =\— that are significant- 
ly biased while other values of x, s away from this disk 
edge rapidly become an unbiased estimator of//. 

The magnitude of the systematic error, x - //, values 
that he on the disk edge of Fig. 5 is shown in Fig. 6(a). 
Also shown in Fig. 6(a) is the lower bound of the 
expected error given by x - I/2. Note that the range of 
errors for a given x results from different //, a points 
mapping to the same value of x, but having different 
systematic errors. Figure 6(b) shows the points in //, a 
space that map to values of x, s that lie on the disk 
boundary; this represents a significant region of //, a 
space and thus gives rise to the density of x, s points on 
the disk boundary shown in Fig. 5. The points shown in 
red in Fig. 6(b) all map very close to the point x = 0, 
5 = 0. 

It is tempting to apply a correction for the average 
systematic error (as a function of x ) to obtain a better 
estimate of //. However, since the bias is nearly a step 



function that rapidly approaches zero away from the 
disk edge, applying such a correction based on the 
proximity of the x, s coordinate to the disk edge is sub- 
ject to misapplication due to statistical variations 
expected in any finite sample size. A simpler but crud- 
er approximation would be to use the uncorrected value 
of X and consider the systematic error to be bounded 
between zero and the line x - 0.5 whenever 5 < 0.6. We 
will consider an uncertainty rule based on this approx- 
imation for the special test scenario. 



4. Errors and Level of Confidence 

For the discrete pdf given by Eq. (1) with standard 
deviation given by Eq. (2) we can explore the relation- 
ship between the standard deviation and the level of 
confidence, i.e., the coverage factor. Although we are 
considering random Gaussian noise corrupting our 
measurement result, the subsequent rounding due to 
finite resolution significantly changes the level of con- 
fidence for a coverage factor of ^ = 2, particularly when 
a < 0.5. We can gain some insight into this effect by 
examining the magnitude of the error at the 95th per- 
centile in relation to the magnitude of the standard 
deviation. The magnitude of the error that occurs at the 
95th percentile is given by the cumulative probability 
distribution associated with Eq. (1) and evaluated to 
determine the integer, 77950/^ , and the corresponding error 
value, £950/^, by increasing the number of terms in Eq. 
(4) until the inequality just holds. 

>v(0,//,cr) + >v(l,//,cr) + H<-1, ^a)+w{2,^d) 

+ ... + >v(7795o/„,//,cr) >0.95 and e,,.,^ ^\n,,.y^-^\ (4) 

Clearly £950^ will depend on the particular values of ji 
and (J, and we expect that £950^ will also display the dis- 
creteness of the underlying discrete pdf For example, 
as a increases with ji fixed, £950^ will remain fixed until 
the inequality in Eq. (4) is violated forcing the addition 
of another term to the cumulative probability and a cor- 
responding jump in the value of £950^. Figure 7(a) shows 
the £^50/^ value over the region: < // < 0.5 and < cr< 1 . 
Figure 7(b) displays the required coverage factor to 
achieve a 95 % level of confidence, i.e., k = £gs%^s. For 
JI small but nonzero and a small, the coverage factor 
approaches infinity since all the measured values of x 
yield zero, resulting in 5 — > while x-ji remains finite. 
One method to avoid infinite coverage factors is to 
establish a finite lower bound for the standard uncer- 
tainty; this is employed by Rules 1 and 2. 
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Fig. 6. (a) the error (x-fj) for points on the disk boundary shown in Fig. 5 together with 
the bounding curve x - Yi shown in blue, (b) The points in jU, a space that map to the disk 
boundary; the red points all map very close to x = 0, ^ = 0. 
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Fig. 7. (a) The 95tli percentile error (in units of resolution) as a function of the true value 
(jX) and the standard deviation of the Gaussian noise (a); (b) the corresponding coverage 
factor required to include the 95th percentile error. 



5, Measurement Process Scenario 

In the measurement process scenario an uncertainty 
evaluation is performed on a measurement system and 
then a future (single observation) measurement result, 
X, which represents the best estimate of the measurand 
is assigned an uncertainty based on the prior evalua- 



tion. Since x is a single reading from an instrument hav- 
ing resolution R, and in this paper we take ^ = 1, hence 
X must be an integer. 

One complicating factor in this scenario is that the 
reproducibility evaluation is conducted on a reference 
artifact having some value //Rgf while the uncertainty 
statement is relevant to a future measurement having a 
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different (infinite resolution) value jU, and the two val- 
ues are independent. For example, consider calibrating 
1 00 mm long artifacts with an instrument having 
0.01 mm of resolution. The reference artifact on which 
the uncertainty evaluation is based might be 100.002 mm 
long, where as the artifacts to be measured in the future 
will each have some other value, e.g., 99.997 mm. 
Hence all of the measurements used in the uncertainty 
evaluation to determine s (based on a large sample of 
measurements) will depend on the particular value of 
jtiRef? ^i^d different values of jtiRef will yield different val- 
ues ofs\ see Fig. 2b. Note that the best estimate of the 
reference artifact, x^ef , is discarded since it provides no 
information about the value of the future measurement 
result. 

In this paper we will examine only two values for 
jtiRef- It is frequently the case that reference artifacts 
used in uncertainty evaluations have a value that is 
exactly on a unit of resolution; we denote this case as 
MRef ^ 0- ^^ example of this would be a digital caliper 
reading in units of inches that is evaluated by examin- 
ing repeated measurements on gauge blocks from an 
English (inch based) set. The caliper may have a reso- 
lution of 0.001 inch and the gauge blocks will be with- 
in a few microinches of some multiple of 0.001 inch. 
(Note for this condition to hold, the systematic error of 
the instrument must be small with respect to the resolu- 
tion.) 

Alternatively, we examine the case where the refer- 
ence artifact has a uniform probability of being distrib- 
uted between and 0.5, we denoted case as j^i^^f = Zj^i. 
In the previous example of the caliper, this could occur 
when repeated measurements are made on several dif- 
ferent metric gauge blocks using an inch based caliper, 
and the results are pooled to obtain the standard deviation. 

We consider three rules to evaluate the standard 
uncertainty due to finite resolution and repeatability in 
the measurement process scenario given in Eq. (5); 



'Ir-" 



Rule 1: u(x) - 



^ 



V y 



+ ^(MRef)' 



Rule 2: u(x) =Max 



2 



'•'(-"Ref) 



Rule 3: u{x) = 



'in' 

2 



Max 






(5) 



where R is the resolution (in this paper R = I), and 
^(i^Ref) is the standard deviation of a large sample of 
measurements on an artifact with a (infinite resolution) 
value jtiRef ■ 

Rules 1 and 2 are based on the GUM and ISO 14253-2 
respectively, as previously described. Rule 3 includes 

the observation that Max[— ^^ ,s]> a. 
VI 2 
All three rules contain two quantities, one dependent 

on R and the other on s, and a prescription to combine 

1 
them. The first quantity (with R= I) has the value -j= 

and represents the standard uncertainty associated with 
a uniform distribution with limits ± Yi, representing the 
fact that the single (future) observation x could be 
located anywhere in the resolution interval. The second 
quantity is an estimate of the underlying population 
variance (7, which gives rise to the observed variation in 
X. That is, the first quantity accounts for the ambiguity 
due to the resolution and the second quantity accounts 
for the reproducibility of the measurement result. 

Rule 1 always uses the calculated standard deviation 
as an estimate for a. However, when a is small, s can 
underestimate (7, e.g., if jU = and a= 0.2 nearly all of 
the repeated observations will be zero and hence ^ ~ 0. 
Rule 2 also estimates a using s and then selects the 
larger of ^ or the standard uncertainty of the resolution. 
We can expect that by omitting one of the two uncer- 
tainty sources, that Rule 2 will underestimate the uncer- 
tainty associated with the measurand. We introduce 
Rule 3 that estimates (7 by selecting a quantity that is an 
upper bound for a, as previously noted in Sec. 3. 



6. Testing the Uncertainty Rules for the 
Measurement Process Scenario 

We can estimate how effective these rules are in pro- 
ducing expanded uncertainty intervals that contain the 
reasonable values that can be attributed to the measur- 
and. In the measurement process scenario, the single 
measurement result, x, is determined by the value of jU 
(which is a property of the object but is unknown), cor- 
rupted by a random perturbation arising from a, and 
rounded to the resolution unit. We can imagine that this 
measurement is repeated a large number of times and 
then ask what fraction of the errors are contained in the 
uncertainty statement, where the error is £ = x - jU, and 
is due to the random Gaussian perturbation and the 
rounding to an integer value. We will adopt this view- 
point since, in commercial metrology, it is typical to 
construct an expanded uncertainty using the coverage 
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factor k = 2 and then expect that 95 % of the potential 
measurement errors will be contained within this 
interval. 

One complicating factor in the measurement process 
scenario is that the probability that a future measure- 
ment result will be contained within the expanded 
{k = 2) uncertainty interval is a function of three popu- 
lation variables: j^^^f, ji, and o. Our approach will be to 
fix the population variables, evaluate the expanded 
uncertainty, U, and then draw a large number of sam- 
ples (each representing a future measurement) and 
determine what fraction are contained within the 
expanded uncertainty interval. We will then select 
another set of population variables and continue this 
methodology until we have examined the set of popu- 
lation variables of interest. 

Figure 8 displays the probability that a future meas- 
urement result will be contained within the expanded 
uncertainty interval evaluated by Rule 1 (the GUM 
rule) with jx^^ = over the domain < jU < 0.5 and < 
cr< 1 . As seen in the figure, the surface consists of a rel- 
atively flat mesa having a containment probability near 
unity, interrupted by abrupt canyons of significantly 
lower containment probability. The corresponding plots 
using Rule 2 and Rule 3 are similar, but with broader 
and deeper canyons, and narrower and shallower 
canyons, respectively. We note that in Fig. 8, and in the 



associated three statistics in Table 1, the containment 
probability only addresses the question of what fraction 
of errors (for a particular jU, a) are contained in the 
uncertainty interval; it does not address their relative 
position within the interval. Hence the same contain- 
ment probability would be assigned to two different 
distributions having the same fraction of uncontained 
errors, despite the fact that one distribution might have 
the uncontained errors only slightly away from the 
uncertainty interval and the other may have its uncon- 
tained errors grossly away from the uncertainty interval. 

Table 1 summarizes these results. Min P is the mini- 
mum containment probability over the domain of pop- 
ulation variables examined. That is, over the domain of 
jU, (lvalues considered, find the particular value of jU, a 
that has the smallest containment probability and report 
that value as Min R For example, when Rule 1 is eval- 
uated with jtiRef = 0, the minimum containment probabil- 
ity occurs at (7= 0.15 and jU = 0.4; that is, for this set of 
values a (randomly chosen) future measurement result 
has only a 0.75 probability of being contained in the 
expanded uncertainty interval of this rule. The Min P 
value tells us something about the lowest point on the 
surface shown in Fig. 8. 

Avg P is the average probability of a measurement 
error containment over all values of jU, a considered; 
for the previous example of Rule 1 with jtiR^f = 0? 



P(X < V{Rlr^Ref = 0)) 




Fig. 8. A plot of the probability that the measurement error of a single observation (hav- 
ing value jd and corrupted by Gaussian noise of standard deviation a) will be contained 
within the expanded uncertainty of Rule 1 evaluated with /J^ef = 0. 
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Avg P(x <U) = 0.97, hence the probability of error con- 
tainment for most values of fi and a is near unity. 

Most metrologists expect approximately 95 % of the 
potential measurement errors to be contained in the 
expanded (k = 2) uncertainty interval. However, even 
for the infinite resolution case of a Gaussian popula- 
tion, 95 % is the expected fraction of error contain- 
ment, and any finite sample size will have slightly 
more, or slightly less, than 95 % of the deviations from 
the mean contained within 2 s. As another measure of 
the effectiveness of an uncertainty rule, we introduce 
the percentage of the surface of Fig. 8 that does not 
include at least a 94 % containment probability; this is 
denoted % P < 0.94. The value of 0.94 was selected 
since, for the selected sample sizes, the corresponding 
infinite resolution case always included at least this 
fraction of the errors. Note both the % P < 0.94 and 
Avg P(x < U) depend on the extent of the //, (7 domain 
under consideration, hence they are relative metrics 
that are only usefiil when comparing rules over the 
same //, a domain. (All values in Table 1 and Table 2 
were generated numerically, and used sample sizes of 
10,000; the uncertainty in the results is estimated at 
0.03 resolution units.) 

From Table 1 we see that none of the three (ad hoc) 
rules guarantees 95 % error containment for all values 
of ji and (7, with Rule 2 failing this criteria 30 % of the 
time for the //Ref = case. All of the three rules benefit- 
ed from computing the standard deviation of the meas- 
urement process using many reference standards with 
different values and pooling the results, i.e., the //Ref = 
SjU case. Indeed, for the //Ref = S// case, Rule 1 (GUM) 
performance approached that of Rule 3 without the 
need to compare the relative size of s to the standard 
uncertainty of the resolution, as required in both Rule 2 
and Rule 3. 

Also shown in Table 1 are three statistics that 
describe the difference between the expanded uncer- 
tainty and the magnitude of the 95th percentile error, 
1^950/J. This tells us something about the amount of the 
over (or under) estimation of the uncertainty interval. 



As might be expected for simple ad-hoc uncertainty 
rules, none of the three rules adapts itself well to the 
discontinuous nature of the discrete pdf, e.g., see Fig. 7, 
and at some values of //, a they significantly overesti- 
mate the magnitude of 1^950/J, while at other points they 
underestimate its magnitude. These statistics will be of 
more value for the special test measurement scenario. 



7. Special Test Scenario 

In the special test scenario, repeated observations are 
used to calculate both the sample mean (which in gen- 
eral will not be an integer) and the standard deviation, 
and these can be expected to approach the discrete pop- 
ulation mean and standard deviation x, s, in the limit of 
large sample size. Hence, in this scenario the number of 
observations used to calculate x, A^^, is the same as the 
number of observations used to calculate the standard 
deviation s. In the special test scenario, see Eq. (7), the 
GUM rule (Rule 1) combines the uncertainty of the res- 
olution unit with the standard deviation of the sample 
mean. For large samples, the uncertainty associated 
with the resolution unit will clearly dominate since the 
standard deviation of the mean scales as \/yJN^. 
Similarly the ISO rule (Rule 2) selects the larger of the 
two terms; hence we expect these two rules to be simi- 
lar since the uncertainty of the resolution unit domi- 
nates the standard deviation of the mean in both cases. 
Consequently it is clear that for large sample sizes both 
of these rules overestimate the uncertainty associated 
with X. 

For Rule 3 we use our observation that near the for- 
bidden disk region (rather crudely described by the 
region ^ < 0.6) x is a biased estimator of ji and the value 
of ji is contained in the interval x < // < 0.5 as seen in 
Fig. 3 . While in principle a correction should be applied 
to account for this bias, in practice the typical user is 
unlikely to accommodate such an inconvenience. 
Consequently we describe the ignorance in the location 
of jU by a uniform distribution as seen in Rule 3 of 



Table 1. Measurement Process Scenario 





Rule 1 (GUM) 


Rule 2 (ISO) 




Rule 3 




oo Resolution 




/^Ref=0 


^Kci=^^ 


/^Ref=0 


^VL^i=^^ 


/^Ref=0 


/^Ref = 


Z^ 


U=2s 


Min Fix < U) 


0.75 


0.84 


0.65 


0.74 


0.85 


0.91 




0.94 


Avg F{x < U) 


0.97 


0.98 


0.95 


0.97 


0.98 


0.98 




0.95 


% P < 0.94 


13% 


7% 


30% 


19% 


10% 


5% 




0.00 


Avg(t/-|£95|) 


0.23 


0.29 


0.12 


0.16 


0.27 


0.30 




0.02 


Max(t/-fe|) 


0.69 


0.89 


0.58 


0.76 


0.81 


0.89 




0.07 


Mm(t/-|£95|) 


-0.13 


-0.25 


-0.38 


-0.48 


-0.15 


-0.14 




0.00 
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Eq. (7). Furthermore, Rule 3 includes the observation 

1 
that Max[~i=, s]> a; outside the disk region Rule 3 

treats the uncertainty evaluation as the infinite resolu- 
tion case. 



Rule 1 : u(x) ■ 



s 



(A 



Rule 2: u(x) =Max 



2 s 



^/3'^/^^ 



Rule 3: u(x) ■ 



^1 ^ 

-R-x 

2 



S 



Max 



s 



, s 



N. 



V / 

s 



-ifs<0.6R 

ifs>0.6R 
(7) 



8. Testing the Uncertainty Rules for the 
Special Test Scenario 

In the case of a large sample size the values of both 
X and s become relatively stable and consequently the 
magnitude of the systematic error \x-fi\ and the uncer- 
tainty rule become stable. Hence we can consider the 
quantity U - \x - ji\ over a region of //, a values to 
determine whether or not a particular uncertainty rule 
contains the systematic error. 

Table 2 displays the information regarding the con- 
tainment probability for the three rules as applied to the 
special test scenario. Both Rule 1 and Rule 2 greatly 
overestimate the uncertainty associated with the mean 
and hence always contain 100 % of the errors. Rule 3 
more closely approaches the 95 % error containment 
interpretation with a minimum containment probability 
of 0.94 and an average (over the //, a values consid- 
ered) of 0.98. 

To consider the magnitude of the overestimation we 
examine the expected values of the difference between 
the expanded uncertainty and \e^s%\' As previously sug- 
gested, Rule 1 and Rule 2 behave similarly and both 
typically overestimate le^^o/j by one-half a resolution 
unit. Rule 3 does significantly better by more closely 



matching the expanded uncertainty to the \£g^o/j value, 
with an average overestimation of 0. 15 resolution units. 

Table 2. Special Test Scenario 



Rule 1 
(GUM) 



Rule 2 
(ISO) 



Rule 3 



Min P(x < U) 


1.0 


1.0 


0.94 


Avg P(x < U) 


1.0 


1.0 


0.98 


% P < 0.94 








1% 


AvgtZ-fel 


0.51 


0.50 


0.15 


Maxt/-|£95| 


0.58 


0.58 


0.58 


Mint/-|£95| 


0.18 


0.18 


-0.01 



In the special test scenario with a large sample size, 

Rule 1 and Rule 2 become identical with u(x) -^ —j^, 
1 _ V12 

— X 
— 2 
and Rule 3 yields u{x) -^ — 1=- ifs < 0.6 and w(x ) ^ 

V3 

\fs> 0.6. Elster [5] has examined the general solution 
for the special test scenario using Bayesian calcula- 
tions. He also considered the special case of data with 
an equal number of ones and zeros (hence x = 0,5 and 
s = 0.25), and showed for large samples that the best 
estimate of ji was the mean x = Yi and u{x ) — > 0; this 
result is given by Rule 3 but not by Rule 1 and Rule 2, 

which yield —j^, 

Vi2 



9. Conclusions 

We have examined the statistics associated with data 
recorded with finite resolution in the limit of large sam- 
ple size. Due to finite resolution, there exists a disk 

^ 2 nt 
+ 5 = — inside which the val- 

. . UJ 

ues of X, s are forbidden. Furthermore, values of x that 
occur on the boundary of the disk are always (except 
for the jU = and // = 0.5 case) systematically biased 
toward the resolution unit. Away from the disk bound- 
ary X rapidly becomes an unbiased estimator of //. 

We have also considered three simple ad-hoc rules 
for evaluating the uncertainty due to finite resolution, 
including those recommended for the by the GUM and 
ISO 14253-2. 

For the measurement process scenario and using the 
criterion that the (k = 2) expanded uncertainty should 
include approximately 95 % of the potential errors, we 
found that the ISO 14253-2 rule underestimates the 
uncertainty for roughly 30 % of the //, (lvalues, while 



defined by 



UC LU ill J 
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the GUM rule and Rule 3 do significantly better. 
Additionally, we pointed out that evaluating the stan- 
dard deviation of repeated measurements using several 
different reference artifacts, with values that are not 
increments of the resolution unit, improves the contain- 
ment probability for all the uncertainty rules examined. 
For the special test scenario, where a large number of 
measurements are available to compute the mean value 
and the standard deviation, both the GUM and ISO 
14253-2 rules greatly overestimate the uncertainty of 
the mean for large sample sizes. This overestimation 
results from setting a lower limit for the uncertainty 

1 
equal to -^= regardless of sample size or the value of 

the standard deviation. Rule 3 more closely matches the 
expanded uncertainty to the 95th percentile error while 
still maintaining a 95 % error containment probability 
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